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Brain  Computer  Interfaces  for  Enhanced  Interaction  with 

Mobile  Robot  Agents 

Principal  Investigator:  Peter  K.  Allen,  Columbia  University 


1  Equipment  Purchased 

1.  PR2  Mobile  Robot,  Willow  Garage,  $175,000.00 

2.  Baxter  Dual-arm  robot.  Rethink  Robotics,  $27,500.00 

2  Research  Projects  Using  the  Equipment 

2.1  Brain  Computer  Interface  for  Grasping  and  Manipulation 

Brain  Computer  Interfaces  (BCIs)  show  great  potential  in  allowing  humans  to  interact  with  compu¬ 
tational  environments  in  a  synergistic  and  complementary  way.  This  project  focused  on  acquiring  a 
mobile  robotic  agent  platform  that  can  be  used  to  explore  these  interfaces  for  a  variety  of  real-world 
tasks  and  environments  of  interest  to  the  Army.  It  aims  to  extend  our  current  work,  for  creating 
systems  enabling  mutually-derived  situation  awareness  which  use  BCI  to  synergistically  couple  hu¬ 
man  expertise  and  computational  intelligence.  This  DURIP  extends  this  work  by  providing  a  test 
environment  where  the  human  control  of  a  robot  agent  can  be  experimentally  validated  in  real-world 
scenarios.  The  robot  platforms  used  are  the  Willow  Garage  PR2  personal  robot  [21],  a  humanoid¬ 
like  robot  with  a  mobile  omnidirectional  base,  and  a  Baxter  dual-aim  robotic  manipulator. 

Brain-Computer  Interfaces  are  promising  technologies  that  can  improve  Human-Robot  Interac¬ 
tion.  Non-invasive  BCI’s,  which  arc  very  desirable  from  a  medical  and  therapeutic  perspective,  arc 
only  able  to  deliver  noisy,  low-bandwidth  signals,  making  their  use  in  complex  tasks  difficult.  To 
this  end,  we  present  a  shared  control  online  grasp  planning  framework  using  an  advanced  EEG- 
based  interface  that  can  interface  with  the  PR2  robot  to  accomplish  Human-in-the  Loop  (HitL) 
grasping  tasks.  Unlike  commonly  used  paradigms,  the  EEG  interface  we  incorporate  allows  online 
generation  of  a  flexible  number  of  options.  This  online  planning  framework  allows  the  user  to  di¬ 
rect  the  planner  towards  grasps  that  reflect  their  intent  for  using  the  grasped  object  by  successively 
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138: 


selecting  grasps  that  approach  the  desired  approach  direction  of  the  hand.  The  planner  divides  the 
grasping  task  into  phases,  and  generates  images  that  reflect  the  choices  that  the  planner  can  make 
at  each  phase.  The  EEG  interface  is  used  to  recognize  the  user’s  preference  among  a  set  of  options 
presented  by  the  planner.  The  EEG  signal  classifier  is  fast  and  simple  to  train,  and  the  system  as  a 
whole  requires  almost  no  learning  on  the  part  of  the  subject. 

Using  EEG  as  the  user  interface  has  a  number  of  advantages.  Firstly,  the  neurological  phenom¬ 
ena  used  in  the  system  is  a  subconscious  reaction  to  visual  stimuli,  and  therefore  needs  very  little 
user  expertise  to  operate.  Secondly,  the  planner  can  take  advantage  of  visual  ambiguity  between 
functionally  similar  grasps  to  achieve  fast  convergence  in  the  shared-control  paradigm.  The  user 
acts  as  a  filter  for  the  planner,  directing  it  to  a  desired  approach  direction  and  filtering  proposed 
candidates  until  a  reasonable  one  is  found.  Three  users  were  able  to  use  this  system  with  minimal 
training  to  pick  up  a  variety  of  objects  in  a  semi-cluttered  scene.  Also,,  five  users  were  able  to  use 
this  system  to  pick  up  objects  in  cluttered  scenes  where  object  recognition  was  unable  to  correctly 
recognize  the  complete  planning  scene,  depending  instead  on  the  user’s  preferences  to  choose  vi¬ 
able  grasps.  These  experiments  suggest  that  the  1)  reduced  cognitive  load  of  the  EEG  version  of  the 
system  allows  the  subjects  to  plan  grasps  in  scenes  far  more  cluttered  than  the  EMG  version  was 
able  to  handle,  2)  substantially  reduces  training  time  over  the  EMG  based  system,  3)  outperforms 
the  previous  version  of  the  EEG  and  EMG  systems  in  terms  of  run-time,  4)  allows  stable  grasping 
amidst  clutter  and  uncertainty,  and  5)  works  across  multiple  hardware  platforms. 


2.2  Experimental  Results 

Figure  1  shows  the  experimental  setup  and  Figure  2  shows  the  grasping  pipeline  we  implemented. 
We  performed  three  experiments  using  a  robotic  manipulator  and  the  PR2-based  system,  using  a 
shared  implementation  of  the  EEG/RSVP  option  selection  and  the  same  training  procedure. 

For  the  first  experiment,  we  validated  the  initial  robot  arm  system  by  asking  three  subjects  to 
instruct  the  robot  to  lift  three  objects  in  a  typical  scene.  The  results  from  this  experiment  show  that 
an  RSVP-based  Human-in-the-loop  planner  can  effectively  be  used  for  grasp  planning. 

In  the  second  experiment,  we  asked  five  subjects  to  use  the  PR2 -based  system  to  lift  the  same 
three  objects  in  a  highly  cluttered  scene,  where  not  all  of  the  objects  in  the  scene  were  in  the  object 
recognition  database.  In  this  experiment,  the  subject  provides  information  that  is  not  available  to  the 
planner:  in  order  to  plan  an  effective  grasp,  the  user  must  ensure  that  the  planner  does  not  choose 
a  grasp  which  would  collide  with  an  unrecognized  object.  Figure  3  shows  this  experimental  setup 
and  Table  1  shows  these  results. 

In  the  third  experiment,  we  asked  five  subjects  to  pick  up  objects  that  were  explicitly  removed 
from  the  object  recognition  database.  In  this  experiment,  we  examine  the  case  where  the  planner 
is  actually  given  an  incorrect  view  of  the  world.  That  is,  we  set  up  the  object  recognition  system 
to  mistakenly  identify  a  shaving  gel  bottle  as  a  shampoo  bottle  which  is  both  wider  and  taller,  so 
a  large  fraction  of  the  suggested  grasps  will  not  lead  to  a  successful  pickup.  In  particular,  grasps 
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(a)  Robot  arm  system 


(b)  PR2  system 


Figure  1 :  (a)  Initial  testing:  The  subject  guiding  a  robotic  arm  through  an  automated  grasping  task  using 
EEG  BCI.  On  the  left  side  is  the  robotic  manipulator  and  three  containers  in  the  grasping  scene.  On  the  right 
is  a  subject  using  the  system  through  the  B-Alert  EEG  cap,  which  is  relatively  unobtrusive  and  can  be  worn 
for  long  periods  of  time.  In  this  example,  at  least  one  of  the  grasps  found  in  the  database  for  the  object  was 
reachable,  and  is  highlighted  in  blue  in  the  upper  left  comer  of  the  grid.  The  user  may  choose  to  execute 
the  highlighted  grasp,  or  to  re-seed  the  planner  with  one  of  the  other  nine  grasps  and  then  re-enter  the  active 
refinement  phase  with  a  new  highlighted  grasp. 

(b)  Implementation  on  PR2  robot:  The  subject  guiding  the  PR2 -based  system  through  the  grasp  selection 
phase.  On  the  far  left  is  the  subject  wearing  the  B-Alert  EEG  cap;  also  on  the  left  is  the  PR2  robot  with  its 
arms  posed  so  as  to  allow  the  Kinect  mounted  on  its  head  a  full  view  of  the  scene.  The  user  is  presented  with 
images  of  grasps  on  the  screen  in  the  far  right.  Once  one  of  the  grasps  is  selected,  the  system  will  proceed  to 
the  grasping  stage. 
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approaching  from  the  top  of  the  shaving  cream  bottle  will  almost  always  fail.  This  experiment 
takes  advantage  of  the  subject’s  ability  to  generalize  grasps,  recognizing  that  a  subset  of  the  grasps 
for  a  mis-identified  object  may  still  work  in  practice.  This  experiment  was  conducted  on  the  same 
five  subjects  as  the  cluttered  scene  experiment  with  the  PR2  during  the  same  test  session.  Thus, 
the  subjects  were  experienced  with  the  use  of  the  RSVP  paradigm  and  had  become  aware  that  the 
planner  would  not  necessarily  suggest  good  grasps  to  the  user.  Table  2  shows  these  results.  All 
testing  was  approved  by  the  Institutional  Review  Board  of  Columbia  University  under  Protocol 
IRB-AAAJ6951. 

2.3  Discussion 

The  results  from  the  two  PR2  experiments  imply  that  there  are  benefits  to  further  leveraging  the 
human’s  ability  to  select  viable  grasps.  For  example,  the  current  system  is  dependent  on  having  a 
set  of  grasps  from  a  pre-computed  grasp  database  for  the  initial  grasp  selection  state.  This  in  turn 
necessitates  that  computer  models  exist  for  all  objects  to  be  grasped.  Ergo,  the  system  cannot  pick 
up  previously  unseen  objects.  However,  recent  work  in  3D  shape  completion  has  shown  that  it  is 
possible  to  plan  grasps  on  completed  meshes,  which  could  result  in  a  system  which  uses  human 
input  to  choose  viable  grasps  for  unknown  objects. 

The  extension  of  the  online  Human-in-the-Loop  planner  to  this  EEG  based  image  streaming 
paradigm  has  just  begun.  In  its  current  implementation,  the  subject  decisions  are  elicited  at  fixed 
points  of  the  pipeline.  Future  work  will  move  towards  attempting  to  integrate  the  EEG  data  in  a 
more  real-time  strategy.  We  are  also  beginning  new  experiments  using  the  Baxter  dual-arm  robot 
that  entail  bi-manual  manipulation  using  the  BCI  interface.  New  research  we  are  proposing  involves 
integrating  a  fNIRS  (Functional  Near-Infrared  Spectroscopy)  with  the  EEG  sensing  to  improve  the 
BCI  response. 

2.4  Papers  and  Videos 

•  Grasping  with  your  Brain:  a  Brain-Computer  Interface  for  fast  grasp  selection  by  Robert 

Ying,  Jonathan  Weisz  and  Peter  K.  Allen,  submitted  to  Int.  Journal  of  Robotics  Research. 

http : //www . cs . columbia.edu/~allen/ ijrr_paper.pdf 

•  Video  of  the  BCI  interface  and  PR2  grasping: 

http : //www . cs . columbia.edu/~allen/IJRR_RSVP.mp4 

•  Video  of  Virtual  Reality  Interface  to  PR2 

https : / /www . youtube . com/watch?v=5aGuwlQu6oc 
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Run  Recognition  Again 


Figure  2:  A  diagram  outlining  the  EEG  RSVP-driven  grasping  pipeline.  In  each  phase,  a  series  of 
images  is  generated  representing  the  available  options,  and  the  user  can  use  the  RSVP  interface  to 
select  an  option. 
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(a)  Scene  from  the  perspective  of  the  PR2-mounted  Kinect  (b)  The  set  of  objects  recognized  by  the  object  recognition 

system 


Figure  3:  (a)  A  cluttered  scene  with  many  objects  and  obstacles  between  the  PR2  gripper  and  the 
objects  in  the  database,  as  seen  from  the  perspective  of  the  Kinect  mounted  on  the  PR2’s  head.  Note 
the  large  wooden  block,  which  blocks  off  a  large  number  of  possible  approach  directions  for  grasps 
involving  the  detergent  bottle. 

(b)  The  same  scene,  seen  from  the  perspective  of  the  object  recognition  system.  In  this  case,  the 
object  database  contains  only  the  detergent  bottle,  the  shaving  gel  bottle,  and  the  shampoo  bottle, 
and  explicitly  excludes  a  model  for  the  wooden  block.  It  is  approximated  with  the  shampoo  bottle 
model,  highlighted  in  green,  which  does  not  fully  represent  the  volume  occupied  by  the  block. 

3  Other  Research  Work 

We  have  also  used  the  Baxter  dual-arm  robot  in  a  number  of  experiments  related  to  grasping  de¬ 
formable  objects.  Grasping  and  manipulation  of  deformable  objects  presents  a  host  of  new  research 
challenges  that  are  much  more  demanding  than  for  rigid  objects.  A  particular  challenge  is  to  fully 
understand  the  physics  of  deformation  and  to  model  deformable  objects  in  a  way  that  can  be  used 
by  real  robotic  systems  in  the  presence  of  noise  and  uncertainty  and  with  real-time  constraints.  This 
research  uses  offline  simulation  to  predict  states  of  deformable  objects  modeled  as  thin-shells  (i.e. 
cloth,  fabric,  clothing)  that  can  then  be  recognized  by  a  robotic  vision/grasping  system  to  pick  up 
and  manipulate  these  objects. 

Papers  resulting  from  this  research  include: 

•  Yinxiao  Li,  Chih-Fan  Chen,  and  Peter  K.  Allen  Recognition  of  Deformable  Object  Category 
and  Pose,  IEEE  Int.  Conf.  on  Robotics  and  Automation(ICRA).  Hong  Kong,  June  2014. 
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•  Yinxiao  Li,  Yan  Wang,  Michael  Case,  Shih-Fu  Chang,  Peter  K.  Allen,  Real-time  Pose  Esti¬ 
mation  of  Deformable  Objects  Using  a  Volumetric  Approach,  IEEE/RSJ  International  Con¬ 
ference  on  Intelligent  Robots  and  Systems  (IROS)  September  14-18,  2014.  Chicago. 

•  Yinxiao  Li,  D.  Xu,  Yonghao  Yue,  Yan  Wang,  S-F  Chang,  E.  Grinspun,  and  P.  K.  Allen. 
Regrasping  and  Unfolding  of  Garments  Using  Predictive  Thin  Shell  Modeling  Video  ,  IEEE 
International  Conference  on  Robotics  and  Automation  (ICRA),  Seattle,  May  2015 

•  Yinxiao  Li,  Y.  Yue,  Danfei  Xu,  Eitan  Grinspun,  Peter  K.  Allen.  Folding  Deformable  Objects 
using  Predictive  Simulation  and  Trajectory  Optimization  IEEE/RSJ  International  Conference 
on  Intelligent  Robots  and  Systems  (IROS)  2015,  Hamburg. 

•  Yinxiao  Li,  Xiuhan  Hu,  Danfei  Xu,  Yonghao  Yue,  Eitan  Grinspun,  and  Peter  Allen,  Multi- 
Sensor  Surface  Analysis  For  Robotic  Ironing,  IEEE  International  Conference  on  Robotics 
and  Automation  (ICRA),  Stockholm,  May  2016. 
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Table  1 :  Experimental  results  from  five  subjects  using  the  PR2  arm  in  a  heavily  cluttered  scene 


Table  2:  Experimental  results  from  five  subjects  using  the  PR2  arm  to  pick  up  mis-identified  objects 


Grasp 

Subject 

Trial 

Misselections 

Iterations 

Successful? 

Time  (s) 

1 

1 

0 

2 

Yes 

90 

1 

2 

0 

2 

Yes 

95 

2 

1 

0 

3 

Yes 

105 

2 

2 

0 

2 

Yes 

95 

3 

1 

0 

3 

Yes 

110 

Mis-identified  Shaving  Gel 

3 

2 

0 

2 

Yes 

100 

4 

1 

0 

2 

Yes 

105 

4 

2 

0 

1 

Yes 

85 

5 

1 

0 

2 

Yes 

100 

5 

2 

0 

4 

Yes 

120 
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